Image coding apparatus with segment classification and segmentation-type motion prediction circuit

ABSTRACT

A segmenting section divides an input image into a plurality of segments. A hierarchizing section determines classes of the respective segments according to a predetermined criterion, and produces a class identification signal indicating the classes of the respective segments. A coding section encodes the segmented image signal into code data while changing the value of a coding control parameter for each of the segments in accordance with the class identification signal. Examples of the coding control parameter are a coding time interval and precision of quantization.

This application is a Divisional of application Ser. No. 11/360,510,filed on Feb. 24, 2006, which is a Divisional of application Ser. No.10/879,192, filed on Jun. 30, 2004; which is a Divisional of applicationSer. No. 09/436,258, filed on Nov. 9, 1999; which is a Divisional ofapplication Ser. No. 08/915,590, filed on Aug. 21, 1997, now abandoned;which is a Divisional of application Ser. No. 08/698,417 filed on Aug.15, 1996, now abandoned. The entire contents of all of theaforementioned applications are incorporated by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image coding apparatus which is usedin image communications equipment such as visual telephone andteleconferencing equipment and image storage/recording equipment such asa digital VTR.

The invention also relates to a motion prediction circuit which performsmotion detection and motion prediction on image information as well asto image coding and decoding apparatuses to which the above motionprediction circuit is applied.

2. Description of the Related Art

Conventionally, various image coding techniques have been developed toimprove the efficiency of transmission and recording of digital images.

FIG. 9 is a block diagram showing a prior art image coding apparatusthat is shown in Recommendation H.261 of ITU-T (International Telegraphand Telephone Consultative Committee). In this apparatus, first an inputdigital image signal 101 to be coded is input to a differentiator 21.The differentiator 21 takes a difference between the input digital imagesignal 101 and a prediction signal 136 (described later), and outputs aresulting difference signal as a prediction error signal 131. A codingsection 22 encodes the prediction error signal 131 and outputs resultingcode data 132. The above Recommendation employs, as a coding method inthe coding section 22, a technique of converting the prediction errorsignal 131 from the spatial domain to the frequency domain by using DCT(discrete cosine transform) that is a kind of quadrature transform, andlinearly quantizing resulting transform coefficients.

The code data 132 as output from the coding section 22 in branched intotwo parts, one of which is sent to a reception-side decoding apparatus(not shown). The other part is input to a decoding section 23 of theimage coding apparatus under discussion. The decoding section 23performs an inverse operation to the operation of the coding section 22,that is, it produces a decoded prediction error signal 133 based on thecode data 132. An adder 24 adds the decoded prediction error signal 133to the prediction signal 136, to thereby produce a decoded image signal134. One frame of the image signal 134 is stored in a memory 25 such asa frame memory. The memory 25 outputs the stored decoded image signal ofone frame after delaying it by one frame, as a decoded image signal 135of a preceding frame. A predictor 26 produces the prediction signal 136and a motion vector 137 by performing motion-compensating predictionbased on the input digital signal 101 to be coded and the 1-framepreceding decoded image signal 135. The motion vector 137 is sent to thereception-side decoding apparatus (not shown) and the prediction signal136 is supplied to the differentiator 21 and the adder 24.

In the conventional image coding apparatus having the aboveconfiguration, the coding section 22 encodes an image of one framesubstantially uniformly irrespective of the content of a subject image.Further, the image 25 operates such that an image of only one frame isstored therein and rewritten every frame. Therefore, the efficiency ofcoding cannot be improved in the conventional image coding apparatus.

A prior art motion prediction circuit is disclosed in JapaneseUnexamined Patent Publication No. Hei. 4-347987, which is shown in FIG.19. In FIG. 19, reference numeral 611 denotes a motion vector detectingcircuit; 612, a segmenting circuit; and 613, a motion parameterdetecting circuit. Further, reference numeral 601 denotes an input imagesignal; 602, interframe motion vectors; 603, a segmentation information;and 604, a motion parameter that is detected on a-segment-by-segmentbasis.

The above circuit operates in the following manner. An input imagesignal 601 is supplied to both the motion vector detecting circuit 611and the segmenting circuit 612. The motion vector detecting circuit 611detects a motion vector 602 on a pixel or small-block basis and suppliesthe detected motion vectors 602 to the motion parameter detectingcircuit 613. On the other hand, the segmenting circuit 612 divides theinput image signal 601 into a plurality of segments having differentmotion by dividing or combining blocks by referring to the motionvectors 602, and outputs segmentation information 603 indicating amanner of segmentation. The motion parameter detecting circuit 613selects motion vectors belonging to a segment from the motion vectors102, and calculates first-order conversion coefficients which describemotion of the segment based on the selected motion vectors. Byperforming this operation for all the segments, the motion parameterdetecting circuit 613 outputs detected motion parameters 604 as motioninformation of the input image.

FIG. 20 shows details of the motion parameter detecting circuit 613.First, coordinates of N measurement points belonging to a target segmentof motion parameter detection are selected. The N sets of measuredcoordinates are supplied to a center-of-gravity detecting circuit 621,which calculates coordinates of the center of gravity of the Nmeasurement points.

The center-of-gravity detecting circuit 621 then converts thecoordinates of the N measurement points into coordinates relative to thecalculated center of gravity, and supplies the resulting coordinates toaverage detecting circuits 622-624. Further, x-components Vx(X, Y) ofthe N measured motion vectors are input to the average detecting circuit622, and y-components Vy(X, Y) are input to the average detectingcircuit 624. The average detecting circuits 622-624 calculate variousaverages and supply those averages to first-order conversioncoefficients detecting circuits 625 and 626. The first-order conversioncoefficients detecting circuit 625 calculates first-order conversioncoefficients a, b, and e by a method of least squares based on thereceived averages. Similarly, the first-order conversion coefficientsdetecting circuit 626 calculates first-order conversion coefficients c,d, and f. These coefficients are used as motion parameters of thesegment concerned. Thus, the motion parameters of each segment of theinput image are obtained.

Another conventional motion prediction circuit is disclosed in JapaneseUnexamined Patent Publication No. Hei. 5-328334. This publicationdiscloses a block coding scheme in which coding is performed in units ofenlarged blocks each obtained by combining similar adjacent blocks.

In the first conventional example described above, a method of leastsquares is used in converting the motion vectors 602 into the motionparameters 604, i.e., first-order conversion coefficients. This is basedon the assumption that the motion of a segment is sufficiently small.Therefore, detection errors become large when the motion is large.

Further, coordinates of N measured points belonging to the targetsegment are needed to produce the motion parameters 604. In the case ofreal-time communication of moving images, this requires that motionvectors of each segment of an input image be measured automatically athigh speed with high accuracy, which is not realistic.

Still further, there is no criterion to judge whether optimum segmentshave been obtained for the motion vectors detecting operation.Therefore, the prediction performance strongly depends on the accuracyof segmentation.

On the other hand, in the second conventional example described above,coding is performed on a block-by-block basis in which a plurality ofblocks are combined into a larger block in a fixed manner. Therefore,the prediction error of an image cannot be made smaller than a block.

SUMMARY OF THE INVENTION

The present invention has been made to solve the above problems in theart and, therefore, has an object of providing an image coding apparatuswhich can improve the efficiency of coding by coding an image inaccordance with its content and rewriting the contents of memories inaccordance with the content of an image.

Another object of the invention is to provide a motion predictioncircuit which can correctly perform motion prediction with a predictionerror smaller than a block, and which can always perform correct motionprediction without the needs of high-speed calculation and obtainingmeasured values.

According to a first aspect of the invention, there is provided an imagecoding apparatus comprising:

-   -   a segmenting section for dividing an input image into a        plurality of segments, and producing an image signal such that        the respective segments are distinguished from each other;    -   a hierarchizing section for determining classes of the        respective segments according to a predetermined criterion, and        producing a class identification signal indicating the classes        of the respective segments; and    -   a coding section for coding the image signal into code data        while changing a value of a coding control parameter for each of        the segments in accordance with the class identification signal.

Further, there is provided an image coding apparatus comprising:

-   -   a segmenting section for dividing an input image into a        plurality of segments, and producing an image signal such that        the respective segments are distinguished from each other;    -   a hierarchizing section for determining classes of the        respective segments according to a predetermined criterion, and        producing a class identification signal indicating the classes        of the respective segments;    -   a plurality of coding sections for coding the image signal into        code data according to different coding schemes; and    -   a switch section for selecting one of the plurality of coding        sections for each of the segments in accordance with the class        identification signal.

Further, there is provided an image coding apparatus comprising:

-   -   a segmenting section for dividing an input image into a        plurality of segments, and producing an image signal such that        the respective segments are distinguished from each other;    -   a hierarchizing section for determining classes of the        respective segments according to a predetermined criterion, and        producing a class identification signal indicating the classes        of the respective segments;    -   a plurality of memories for storing a past image signal;    -   a prediction section for reading, on a segment-by-segment basis,        a prediction source image signal that is similar to the image        signal produced by the segmenting section from the plurality of        memories, and for producing a prediction signal by predicting a        current-frame image signal on a segment-by-segment basis based        on the prediction source image signal;    -   a prediction error extracting section for extracting a        prediction error signal on a segment-by-segment basis based on        the prediction signal and the actual image signal of a current        frame that is produced by the segmenting section;    -   a coding section for coding the prediction error signal into        code data;    -   a decoding section for decoding the coded data into a decoded        prediction error signal;    -   a signal combining section for combining the decoded prediction        error signal and the prediction signal into a decoded image        signal; and    -   a memory control section for storing portions of the decoded        image signal corresponding to the respective segments into the        plurality of memories while switching among the plurality of        memories in accordance with the class identification signal, to        thereby update storage contents of the plurality of memories.

According to a second aspect of the invention, there is provided asegmentation-type motion prediction circuit comprising:

-   -   first segmenting means for segmenting, when necessary, each of        blocks of an input image signal into a plurality of segments by        extracting a feature of the input image signal on a        block-by-block basis;    -   motion vector detecting means for detecting motion vectors for        the respective segments;    -   segment boundary line approximating means for determining a        pattern and a level of each of the blocks from a boundary line        between the segments belonging to the block;    -   motion vector re-search means for adjusting motion vectors of        the respective segments based on the pattern and the level, and        for producing segment boundary line information, final motion        vectors, and prediction image information.

There is provided an image coding apparatus comprising:

-   -   a segmentation-type motion prediction circuit comprising:    -   segmenting means for segmenting each of blocks of an input image        signal into a plurality of segments by extracting a feature of        the input image signal;    -   motion vector detecting means for detecting motion vectors for        the respective segments;    -   segment boundary line approximating means for determining a        pattern and a level of each of the blocks from a boundary line        between the segments belonging to the block; and    -   motion vector re-search means for determining motion vectors of        the respective segments based on the pattern and the level,        producing prediction image information, and transmitting the        final motion vectors and segment boundary line information;    -   quantizing means for quantizing transform coefficients that is        obtained by transform-coding a difference between the input        image signal and the prediction image information;    -   dequantizing means for dequantizing the quantized transform        coefficients;    -   a frame memory for storing, as a decoded image signal of a        current frame, a sum of a decoded image signal of a preceding        frame and a prediction error image signal that is obtained by        inverse-transform-decoding the dequantized transform        coefficients; and    -   variable-length coding means for coding the quantized transform        coefficients and transmitting resulting coded image data.

There is provided a segmentation-type motion-predictive image decodingapparatus comprising:

-   -   dequantizing means for dequantizing decoded transform        coefficients of transmitted or reproduced coded image data;    -   inverse transform decoding means for inverse-transform-decoding        the dequantized transform coefficients into a prediction error        image signal;    -   segment boundary line decoding means for producing a boundary        line of each block from a transmitted or reproduced segment        boundary line information; and    -   prediction image generating means for generating a prediction        image signal from the boundary line, transmitted or reproduced        motion vectors of respective segments, and a decoded image        signal of a preceding frame; and    -   means for producing a decoded image signal from the prediction        error image signal and the prediction image signal.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing an image coding apparatus according toa first embodiment of the present invention;

FIG. 2 shows an example of an input image that is given to an imagecoding apparatus;

FIG. 3 illustrates sub-sampling;

FIG. 4 is a block diagram showing an image coding apparatus according toa second embodiment of the invention;

FIG. 5 is a block diagram showing an image coding apparatus according toa third embodiment of the invention;

FIGS. 6A-6C illustrate contents of memories used in the thirdembodiment;

FIG. 7 is a block diagram showing a modification of the third embodimentin which the contents of one of the memories are not updated;

FIGS. 8A and 8B illustrate extrapolation performed on a similar imagesignal as read out from one of the memories in the third embodiment;

FIG. 9 is a block diagram showing a conventional image coding apparatus;

FIG. 10 is a block diagram showing the configuration of asegmentation-type motion prediction circuit according to a fourthembodiment of the invention;

FIG. 11 illustrates operations of a segment combining section, a segmentboundary line approximating section, and a motion vector re-searchsection in the fourth embodiment;

FIG. 12 shows patterns retained by the segment boundary lineapproximating section in the fourth embodiment;

FIGS. 13A and 13B illustrate a relationship between segment boundaryline coded information and a segment boundary line pattern and level inthe fourth embodiment;

FIG. 14 shows the configuration of a segmenting section of a motionprediction circuit according to a fifth embodiment of the invention;

FIG. 15 shows the configuration of a motion prediction circuit accordingto a sixth embodiment of the invention;

FIG. 16 shows the operation of sections downstream of a segmentingsection in the sixth embodiment;

FIG. 17 shows the configuration of an image coding apparatus to aseventh embodiment of the invention to which the segmentation-typemotion prediction circuit of the fourth embodiment is applied;

FIG. 18 shows an image decoding apparatus according to an eighthembodiment of the invention;

FIG. 19 is a block diagram showing the configuration of a conventionalmotion prediction circuit; and

FIG. 20 is a block diagram showing the configuration of a motionparameter detecting circuit used in the conventional motion predictioncircuit of FIG. 19.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Embodiments of the present invention will be hereinafter described withreference to the accompanying drawings.

Embodiment 1

FIG. 1 is a block diagram showing an image coding apparatus according toa first embodiment of the invention. As shown in FIG. 1, first an inputdigital image signal 101 to be coded is input to a segmenting section 1.Having an image storage memory such as a frame memory, the segmentingsection 1 develops an input image in the memory based on the input imagesignal 101 of one frame, and dividing the thus-developed input imageinto a plurality of segments by utilizing, for instance, a particularshape included in the image.

An example of a segmentation technique is such that edges of an inputimage are detected by, for instance, differentiating the input imagedeveloped on the image storage memory and a portion of the input imageenclosed by the edges is made one segment. For example, where theapparatus under discussion is applied to teleconferencing, an imageincluding persons and a background as shown in FIG. 2 is generallyprovided as an input image. By segmenting this image by edge detection,the image can be divided into a segment 201 of a moving person, asegment 202 of a motionless person, a segment 204 of a desk, a segment204 of a painting, a segment 205 of a clock, and a segment 206 of a wall(the segments 203-206 constitute the background and the segments 201-203constitute foreground).

In performing segmentation by edge detection, as for, for instance, animage of a human face, a segment of eyes, a nose, a mouth, etc. eachbeing enclosed by edges are detected in a segment of the entire facewhich is also enclosed by edges. Such internal segments may be handledtogether as an integral segment of the entire face, rather than asseparate segments. This contributes to improvement of coding, becausethe small individual segments need not be processed in the codingoperation. The integration of the internal segments may be performed bya pattern recognition technique, for instance. That is, when a certainsegment is recognized as representing a face by pattern recognition orthe like, segments located therein and each enclosed by edges areintegrated into a single segment rather than left separated.

After completion of the segmentation, the segmenting section 1 outputsan image signal 102 that is the input digital image signal 101 plussegmentation information, such that data of respective segmentsconstituting the image data on the memory of the segmenting section 1are output sequentially. In the case of the image of FIG. 2, data of therespective segments are output sequentially such that data of thesegment 201 (moving person) is output first, data of the segment 202(motionless person) is output next, and so forth, as the image signal102. Each segment signal of the image signal 102 includes addressinformation indicating the position each segment signal on the screen.Alternatively, the image signal 102 may be produced by outputting theimage in the memory of the segmenting section 1 from its top-leftportion to its bottom-right portion in the order of scanning lines, asin the case of the input digital image signal 101. In this case,identification information indicating a segment to which each componentsignal belongs is added to the component signal overhead information.

The thus-obtained image signal 102 including the segmentationinformation is branched into two parts, which are respectively input toa hierarchizing section 2 and a coding section 4.

Having an image storage memory such as a frame memory as in the case ofthe segmenting section 1, the hierarchizing section 2 develops images ofthe respective segments in the memory based on the image signal 102 thatis supplied from the segmenting section 1. The hierarchizing section 2judges degrees of importance of the respective segments of the entireimage based on their developed images, and classifies the respectivesegments in accordance with their degrees of importance.

Where the respective segment signals of the image signal 102 is producedsequentially, the images of the respective segments are sequentiallyformed on the memory of the hierarchizing section 2 and subjected, inthe same order, to the degree-of-importance judgment, whereby they areclassified. On the other hand, where the image signal 102 is produced inthe ordinary scanning order with overhead information indicating acorresponding segment added to each component signal, the degree ofimportance of the respective segments is judged after the entire imageis formed on the memory of the hierarchizing section 2.

The degree of importance of an image segment increases as the degree ofattention of a viewer to that image segment in the entire imageincreases. However, the standard of judgment depends on an applicationof the apparatus concerned. For example, in the case ofteleconferencing, persons are most important and still objects such as adesk/and a wall which constitute the background and foreground are lessimportant. Further, it can be said that a person who is speaking (i.e.,moving) is more important than motionless persons. Therefore, in theexample of FIG. 2, where the respective segments are to be classifiedinto three classes, the classification may be performed such that thesegment 201 (moving person) is classified as class 1 (highest degree ofimportance), the segment 202 (motionless person) is classified as class2 (medium degree of importance), and the segments of the desk, wall,etc. are classified as class 3 (lowest degree of importance). Otherexamples of indices representing the degree of importance are themagnitude of motion of a segment and the frequency of motion of asegment. In such cases, the degree of importance increases as the motionof a segment is larger or more frequent. The standard to be used injudging the degree of importance is set properly in accordance with, forinstance, an application of the apparatus concerned.

For example, to perform the above-described segmentation by edgedetection, the hierarchizing section 2 distinguishes segments of personsfrom the other segments such as a background wall by performing, forinstance, pattern recognition on edge shapes of the respective segments,and further distinguishes a person who is speaking from the otherpersons by considering additional information on the motion of eachsegment between the current and preceding frames. The classification isperformed such that a segment of a person who is speaking is classifiedas class 1 (highest degree of importance) and segments of objectsexcluding people are classified as class 3 (lowest degree ofimportance).

The hierarchizing section 2 classifies the respective segments in theabove manner, and supplies a class identification signal 103 indicatingclasses of the respective segments to the coding section 4.

The coding section 4 encodes the image signal 102 that is output fromthe segmenting section 1 and includes the segmentation information byPCM, DPCM, or the like, and outputs code data 105. During this codingoperation, the coding section 4 receives the class identification signal103 of the respective segments from the hierarchizing section 2 insynchronism with the image signal 102. The coding section 4 encodes theimage signal 102 of the respective segments while changing the value ofa coding control parameter in accordance with the class identificationsignal 103.

An example of the coding control parameter is the time interval ofcoding. In this case, the coding time interval is set longer for a classhaving a lower degree of importance in accordance with the classidentification signal 103 that is supplied from the hierarchizingsection 2. That is, the coding time interval is increased, that is, thecoding frequency is reduced as the degree of importance of a segmentdecreases. For example, a signal of a segment of class 1 (highest degreeof importance) is coded every frame, a signal of a segment of class 2(medium degree) is coded once per several frames, and a signal of class3 is coded once per several tens of frames. By virtue of this operation,the amount of code data generated can be reduced from the case ofuniform coding over the entire screen, that is, the coding efficiencycan be improved. The coding time intervals for the respective classesare determined properly in consideration of a tradeoff between the imagequality and the coding efficiency.

In teleconferencing etc., there may occur a case that behavior of eachperson is the only information that needs to be transmitted as an imageand images of the background etc. are not necessary. In this case, itmay be conceivable to code only the signals of class-1 and class-2segments (persons) and stop coding of class-3 segments (backgroundetc.), thus enabling further reduction of the amount of code datagenerated. In this case, the decoding apparatus cannot obtain code dataof segments of the background etc. A decoded image is produced byoutputting, for the segments of the background etc., a blue-back signalor a stored background image, for instance.

An example of the coding control parameter other than the coding timeinterval is the precision of quantization of the coding performed in thecoding section 4. That is, based on the class identification signal 103,the coding is so performed so that the precision of quantization is sethigher for segments of a higher degree of importance, thus reducing theamount of code data generated. With this operation, the amount of codedata generated can be reduced as a whole while high image quality ismaintained in segments having a high degree of importance.

Another example of the coding control parameter is the sub-samplingratio utilized in coding the image signal 102, which is described withreference to FIG. 3. In FIG. 3, circles A1-D4 represent pixels in animage. In coding a segment of a class of a high degree of importance,all the pixels A1-D4 (image signal 102) are coded. Conversely, in codinga segment of a class of a low degree of importance, the pixels aresub-sampled every other pixel in both horizontal and verticaldirections: pixels A1-A4 are coded while pixels B1-B4, C1-C4 and D1-D4are not. By increasing the degree of thinning of pixels (i.e.,sub-sampling ratio) for a segment having a lower degree of importance,the amount of generated code data of the entire image can be reduced.The degree of thinning is set properly in accordance with the degree ofimportance of the respective classes.

A further example of the coding control parameter is the threshold ofthe amount of code data generated. That is, in the coding section 4, athreshold (upper limit value) of code data generated is set for eachclass and control is so performed that coding is stopped when thegenerated code amount has reached the threshold. For example, thethreshold of code data generated is set larger for a class of a higherdegree of importance to allow coding of the entire image signal 102.Conversely, the threshold is set smaller for a class of a lower degreeof importance and the coding is stopped when the amount of code datagenerated has reached the threshold. That is, the threshold is setsmaller for a class of a lower degree of importance. As a result, theamount of generated code data of the entire image can be reduced whilehigh image quality is maintained in segments having a high degree ofimportance.

In this case, the decoding-side apparatus cannot obtain image data afterthe coding is stopped. The entire image is produced, for instance, byusing a corresponding image portion of the preceding frame.

As described above, according to the invention, an image for coding issegmented, resulting segments are classified in accordance with theirdegrees of importance, and the value of the coding control parameter ischanged in accordance with the class of a segment in coding an imagesignal portion of that segment. Thus, the coding efficiency of theentire image can be improved by reducing the amount of generated codedata of segments having a low degree of importance.

While the above description of the embodiment is directed to the casewhere the segmentation is performed by using edges in an image and theclassification (hierarchization) is performed based on edge shapes, thesegmentation and classification may be performed in a different manner.

For example, the segmentation and classification may be performed byusing texture analysis, which is similar to the edge detection. In thetexture analysis, an image is divided into segments each having acertain structure (texture), such as a segment of vertical stripes and asegment of horizontal stripes. Since in many cases a wall, a desk, etc,have a certain structure (pattern), the texture analysis enablessegmentation into a segment of the background (wall etc.) and theremaining segment. The segments thus obtained are classified byperforming pattern recognition on their shapes or textures.

The edge detection and the texture analysis are techniques for dividingan image into segments of varied shapes each being enclosed by anoutline (width: one pixel (or several pixels at maximum)) that dependson the content of an enclosed image. However, the invention can also beapplied to a case of dividing and processing an image including blocksof the same size (the block division itself is conventional). In thiscase, a blocking section is provided upstream of the segmenting section1, and an image signal is divided into blocks of the same size (forinstance, 8.times.8 pixels) in the blocking section. The segmentingsection 1 combines, into an integral segment, blocks having similarimage signal portions. The hierarchizing section 2 performsclassification by judging, for instance, whether each segment representsa person or the background by performing pattern recognition on theshape, pattern, or the like of each segment. Where an image is dividedinto blocks as in this case, a block-based coding technique such as DCTcan be used.

The segmentation and the classification may also be performed based onthe activity of an image signal. The activity means the degree ofvariation of an image signal, and may be a squared error from theaverage of an image signal, that is, a variance. More specifically, animage is divided into blocks of the same size (for instance, 8.times.8pixels), an average of an image signal portion and a variance from theaverage are calculated for each block, and the segmentation orclassification is performed based on the average and the variance in amanner as exemplified below. Since in many cases an image portion of awall or the like of the background has a fixed pattern, blocks havingthe same average and variance are regarded as a wall or the like andcombined into a single segment, and classified as a class of a lowdegree of importance. Further, since a block including an edge such asan outline of a person has a large variance, i.e., activity, thesegmentation is performed such that a portion enclosed by blocks havinga large activity are regarded as a single segment. A judgment as to, forinstance, whether that segment is actually a segment of a person is madeby performing pattern recognition on its shape or the like, and theclassification is performed based on the judgment.

Further, the segmentation and the classification may also be performedby using image data of past frames, that is, by utilizing the temporalinformation. That is, image data of past frames are stored in thememory, and motion of each portion of an image is detected by comparingan image of the current frame with images of the past frames. The imageis divided into segments with motion and still segments. The formersegments are classified as segments having a high degree of importanceand the latter segments are classified as segments having a low degreeof importance. Instead of segmenting an image merely based on theexistence of motion, portions having approximately the same magnitude ofmotion may be combined into a single segment instead of segmenting animage merely based on the existence of motion. In this case, forexample, a segment having a larger magnitude of motion is classified asa class of a higher degree of importance.

In the case of segmenting an image in accordance with its motion,segments may be classified in accordance with the frequency of motionrather than the magnitude of motion. More specifically, thehierarchizing section 2 stores motion information over a plurality offrames, calculates a frequency of motion of each segment covering aplurality of frames, and a segment having a higher frequency of motionis classified as a class of a higher degree of importance. By using ahistory of motion over a plurality of frames, a segment that istemporarily motionless can be classified without unduly lowering itsdegree of importance.

Embodiment 2

FIG. 4 is a block diagram showing an image coding apparatus according toa second embodiment of the invention.

While in the first embodiment the value of the coding control parameterin the coding section 4 is changed in accordance with the class of asegment being coded, in the second embodiment a plurality of codingsections of different coding schemes are used and one of the codingsections is selected in accordance with the class of a segment.

In FIG. 4, the segmenting section 1 and the hierarchizing section 2operate in the same manner as in the first embodiment.

Coding sections 4 a-4 c employ different coding schemes. For example,the coding section 4 a performs coding which generates a large amount ofcode data but allows production of a lossless decoded signal. The codingsection 4 b performs coding which reduces the amount of generated codedata to some extent by conducting quantization, for instance. The codingsection 4 c performs very rough coding in which, for instance, only anaverage of pixel levels of each segment is output.

In coding each segment, a class identification signal 103 is supplied toa switch section 6 in synchronism with a portion-of an image signal 102corresponding to the segment. In the switch section 6, switching isperformed in accordance with the class identification signal 103 so thata portion of the image signal 102 corresponding to each segment is inputto a coding section suitable for a class of the segment. For example, asegment of the image signal 102 of a moving person or the like whichsegment is classified as a class of a highest degree of importance iscoded by the coding section 4 a which allows the decoding side toproduce a high-quality image, and code data 105 a is obtained. A segmentof a motionless person which segment is classified as a class of amedium degree of importance is coded by the coding section 4 b, and codedata 105 b is obtained. Further, a segment of the background or the likewhich segment is classified as a class of a low degree of importance andwhich permits generation of a decoded signal that is much different froman input signal is coded by the coding section 4 c, and code data 105 cis obtained.

In this manner, this embodiment can improve the coding efficiency byreducing the amount of generated code data of segments having a lowdegree of importance.

Although this embodiment is directed to the case where segments areclassified into three classes, the number of classes is not limited tothree. Where segments are classified into some other number of classes,there may be provided coding sections of different coding schemes in thesame number as the number of classes.

It is not always necessary that the number of coding sections be equalto the number of classes. That is, the number of coding sections can bereduced by partially employing the operation of the first embodiment inwhich the value of the coding control parameter is changed. For example,two coding sections are provided for three classes such that one of thetwo coding section operate for two classes and the value of the codingcontrol parameter is changed in accordance with the class in that codingsection. This configuration provides the same advantage as describedabove.

The coding schemes employed in the coding sections 4 a-4 c are notlimited to the above ones, but may be selected for the respectiveclasses in accordance with the quality of a decoded image required foreach class and an allowable amount of generated code data for eachclass.

Embodiment 3

FIG. 5 is a block diagram showing an image coding apparatus according toa third embodiment of the invention.

In this embodiment, the control in accordance with the class of eachsegment is applied to interframe-prediction-based coding. As is known inthe art, in interframe-prediction-based coding, a prediction signal isgenerated by predicting an image of the current frame from an image of apast frame stored in a memory, a prediction error signal is calculatedfrom the prediction signal and an actual input image signal by, forinstance, taking a difference therebetween, and the prediction errorsignal is encoded and output. On the decoding side, a similar apparatusfor calculating a prediction signal is provided, and a decoded signal isproduced by adding the prediction error signal that is sent from thecoding side to the prediction signal.

In FIG. 5, the segmenting section 1 and the hierarchizing section 2operate in the same manner as in the first embodiment.

In this embodiment, memories (for instance, frame memories) 10 a-10 cstore images of respective classes. That is, the configuration of FIG. 5is such that segments are classified into three classes. For convenienceof description, it is assumed that the memories 10 a-10 c are forsegments of the class of a highest degree of importance, the class of amedium degree of importance, and the class of a lowest degree ofimportance, respectively. In this case of coding, for instance, theimage of FIG. 2, data of the segment 201 (moving person) is stored inthe memory 10 a, data of the segment 202 (motionless person) is storedin the memory 10 b, and data of the desk segment 203, the paintingsegment 204, the clock segment 205, and the wall segment 206 whichconstitute the background and the foreground are stored in the memory 10c (see FIGS. 6A-6C).

The contents of the memories 10 a-10 c are images obtained by coding andthen decoding an input image signal, i.e., images of a past frame. Thecontents of the memories 10 a-10 c are updated every frame or once perseveral frames (described later).

In coding a portion of an image signal 102 corresponding to eachsegment, which signal is output from the segmenting section 1, aprediction section 12 searches the memories 10 a-10 c for an imagesignal portion similar to the above portion of the image signal 102.More specifically, in this embodiment, the segmenting section 1sequentially outputs portions of the image signal 102 corresponding torespective segments so that a portion of each segment is output en bloc.The prediction section 12 finds an image portion similar to the segmentto be coded from the image data (image signal portions) stored in thememories 10 a-10 c by comparing the integral image signal portion of thesegment with the image data of the respective segments stored in thememories 10 a-10 c. For example, this is done by calculating differencesbetween the portion of the image signal 102 corresponding to the segmentto be coded and the respective image data in the memories 10 a-10 c, andemploying, as a similar image portion, image data in the memories 10a-10 c which provides the smallest difference.

A similar image signal thus obtained by searching the memories 10 a-10 cis used as a prediction source image signal in the prediction section12. That is, the prediction section 12 produces a prediction signal 112by performing prediction based on the prediction source image signal byusing a given prediction function. This operation is performed for eachsegment obtained by the segmenting section 1. The prediction signal 112is output to a prediction error extracting section 14 and a signalcombining section 17. Further, at the same time as the output of theprediction signal 112, the prediction section 12 outputs, to thedecoding side, signal 114 indicating the number of a memory where aprediction image portion that was used for the prediction was stored andan address of the similar image portion on the memory. The decoding sideis provided with a prediction section, three memories, etc. that aresimilar to those in this image coding apparatus. The signal 114indicating the memory number and the address is used to generate aprediction signal in the prediction side.

The prediction error extracting section 14 extracts, for each segment, aprediction error signal 116 based on the image signal 102 and theprediction signal 112. This may be done by simply calculating adifference between the image signal 102 and the prediction signal 112,or by calculating a difference after applying proper weights to bothsignals.

A coding section 4′ encodes the prediction error signal 114 according toa certain coding scheme, and outputs code data 105′.

A decoding section 16 decodes the code data 105′ as output from thecoding section 4′ by performing an inverse operation to the operation ofthe coding section 4′, and outputs a decoded prediction error signal120. The signal combining section 17 combines the decoded predictionerror 30 signal 120 with the above-mentioned prediction signal 112, toproduce a decoded image signal 122. This may be done by simply addingtogether the two signals, by adding together the two signals afterapplying weights thereto, or by other methods. One of those methods isemployed depending on the method of extracting the prediction errorsignal 116 in the prediction error signal extracting section 14.

The decoded image signal 122 is input to a memory control section 18.The memory control section 18 switches among the memories 10 a-10 c inaccordance with a class identification signal 103 supplied from thehierarchizing section 2 so that portions of the decoded image signal 122corresponding to the classes of respective segments. The storage data ofthe memories 10 a-10 c are updated every time they receive the decodedimage signal 122.

The memory (in this embodiment, memory 10 c) for storing motionlesssegments such as the background may be of a type in which newly inputdecoded image signal portions are overwritten on the storage data ratherthan simply substituted for the latter.

For example, in the image of FIG. 2, portions of the background appearor hide behind the person 201 as he moves. As the contents of the memory10 c for storing image data of the respective background segmentscontinue to be updated, the area of the background covered by the imagedata stored in the memory 10 c gradually increases. Therefore, considera case where in a certain frame a movement of the person 201 has causeda portion of the background which has hidden behind the person 201 toappear. In this case, the above configuration provides an advantage thatif the decoding side has the same memory contents as this codingapparatus, transmission of mere address information of the backgroundportion that has newly appeared allows the decoding side to produce animage by reading image data of the wall and the painting from thememory. Once image data of the entire background has been stored in bothmemories of the coding side and the decoding side, it is no longernecessary for the coding side to send code data and it suffices for thecoding side to merely send address information. Thus, the amount ofgenerated code data gradually decreases.

In the above embodiment, the coding section 4 codes portions of theprediction error signal 116 corresponding to all the segments by usingthe coding control parameter of the same value. Instead, as in the caseof the first embodiment, the value of the coding control parameter ofthe coding section 4 may be changed in accordance with the classidentification signal supplied from the hierarchizing section 2. Thisenables further increase of the coding efficiency.

While in the above embodiment only one coding section is provided, thecoding apparatus may be so modified that a plurality of coding sectionsof different coding schemes are provided as in the second embodiment anda proper one of the coding sections is selected for each segment inaccordance with the class identification signal 103 supplied from thehierarchizing section 2. In this case, where a coding section for codingsegments of a certain class is of a type that does not calculate aprediction error, no corresponding memory is necessary because noprediction is performed.

In the above embodiment, in generating the prediction signal 112 theprediction section 12 searches all the memories 10 a-10 c to find imagedata that is similar to each segment of the input image signal 102 ofthe current frame. Instead, the coding apparatus may be so modified thatthe prediction section 12 searches only the memory corresponding to theclass of each segment in accordance with the class identification signal103 that is supplied from the hierarchizing section 2. This facilitatesthe search for similar image data, thus contributing to increase of thecoding efficiency.

While in the above embodiment the memory contents are rewritten everytime the coding section 4 and the decoding section 16 encodes or decodesa signal, the coding apparatus may be so modified that the memorycontents are rewritten every preset time interval. In this case, thecontents of a memory corresponding to class of a higher degree ofimportance are rewritten at a higher frequency, i.e., at shorter timeintervals. This accelerates the memory updating processing.

In this modification, where no background information is necessary forteleconferencing use or the like, the interval of rewriting the memoryfor storing background information may be set at infinity, that is, thememory rewriting may be omitted. In this case, the memory for storingbackground information needs to store it in advance. For example,background information may be stored in the memory before ateleconference is started. FIG. 7 is a block diagram showing an imagecoding apparatus having such a configuration. This apparatus isdifferent from the apparatus of FIG. 5 in that the decoded image signal122 is not input to the memory 10 c.

In the above embodiment, in the predicting operation only image datasimilar to an input image signal portion is found from the memories andused for the prediction. Where the above embodiment is applied to thestandard called MPEG1, a Recommendation of MPEG (Moving Picture ExpertsGroup) of ISO, the coding apparatus may be so modified that a pluralityof similar image data are read from the memories and combined together.MPEG1 employs a procedure in which consecutive frames are coded in anorder different than at the time of input, for instance, in the order ofthe first frame, the fourth frame, the second frame, the third frame,and so forth. In this case, image data similar to those of the secondand third frames can be obtained from image data of the first and fourthframes by averaging, for instance. By performing the prediction based onthe image data thus obtained, the power of the prediction error signaland, therefore, the amount of generated code data can be reduced. Thecoding apparatus of the above embodiment can be applied to theabove-mentioned standard if each of the memories 10 a-10 c can storeimage data of a plurality of frames on a frame-by-frame basis.

In the above embodiment, when the shape or size of a segment of aprediction source image signal as read from the memories 10 a-10 c isdifferent from that of a segment of the coding target image signal 102,the prediction section 12 may perform prediction after the shape or sizeof the segment of the prediction source image signal is changed byinterpolation or extrapolation so as to coincide with that of thesegment of the image signal 102. Since the prediction source imagesignal and the image signal 102 are deviated from each other by oneframe (or several frames), even segments of those signals representingthe same object may different in shape or size. If the predictionsection 12 performed prediction by using a prediction source signalhaving a different shape or size, a resulting prediction error signalwould have large power, leading to increase in the amount of code datagenerated. To solve this problem, the prediction section 12 performsprediction after the shape or size of the segment of the predictionsource image signal is changed by interpolation or extrapolation so asto coincide with that of the segment of the image signal 102.

FIGS. 8A and 8B illustrate how extrapolation is performed. The outlineof a segment of the coding target image signal 102 is indicated by asolid line in FIG. 8B, and the −35 outline of a segment of acorresponding similar image signal (prediction source image signal)stored in one of the memories 10 a-10 c is indicated by a solid line inFIG. 8A. In the figures, hollow circles and sold circles representpixels and, in particular, the solid circles represent pixels includedin the segment of the similar image signal that is stored in the memory.In this case, prediction with extrapolation is performed becauseinformation of hatched pixels of FIG. 8B cannot be obtained directlyfrom the solid-circle pixels. A simple method of effecting suchextrapolation is to give the hatched pixels the same values as those ofsolid-circle pixels located at the boundary. Alternatively, the hatchedpixels may be given values that are proportional to their distances fromthe center of the segment. While the above description relates to thecase of expanding a segment of a similar image signal, reduction of asegment is performed in a similar manner. In this manner, by performingthe prediction after the shape or size of a segment of a predictionsource image signal is changed by interpolation or extrapolation so asto coincide with that of a coding target segment, the prediction errorsignal 116 can be reduced in magnitude, thus enabling reduction in theamount of code data generated.

Embodiment 4

The configuration and operation of a motion prediction circuit accordingto the fourth embodiment of the invention will be described below.

FIG. 10 shows the configuration of this motion prediction circuit. InFIG. 10, a segmenting section 51, a segment boundary line approximatingsection 54, and a motion vector re-search section 55 are novelcomponents. Reference numeral 53 denotes a segments combining section. Amotion vector detecting section 52, a segment boundary line codingsection 56, and a frame memory 57 are conventional components.

Reference numeral 501 denotes an input image signal; 502, segment imageinformation; 503, combined segment image information; 504, approximatedsegment image information; and 505-507, prediction segment imageinformation stored in the frame memory 57. Further, reference numeral508 denotes segment boundary line information; 509, segment boundaryline coded information (novel feature of the invention); 510, motionvectors (novel feature); 511, control information; and 534, predictionblock image information.

The operation of the above motion prediction circuit will be describedbelow.

Referring to FIG. 10, an input image signal 501 is supplied to thesegmenting section 51. The segmenting section 51 divides the inputimage, which includes blocks, into segments, and outputs segment imageinformation 502. Conventionally, there have been proposed varioustechniques of dividing one block into a plurality of segments whichtechniques utilize the brightness, the color difference, the edgeinformation, etc. In this embodiment, the segmentation is performed byusing one those techniques.

FIG. 11 illustrates the operations of segments combining, segmentboundary line approximation, and motion vector re-search which areperformed in this embodiment.

In this embodiment, although the segmentation itself is conventional, itis a first important feature that the segmentation is performed suchthat an image of each block is further divided. The segmenting section51 divides a block image into, for instance, three segments A, B and Cas shown in part [I] of FIG. 11. Receiving the segment image information502, the motion vector detecting section 52 detects a motion vector 512for each of the segments A, B and C by referring to coded imageinformation 505 that is stored in the frame memory 57. The motion vectordetection itself is a conventional technique. That is, a predictionsegment image having a smallest error with respect to an input segmentimage is found from the coded images, and the position of a foundprediction segment image relative to the input segment image is employedas a motion vector.

Next, the operation of the segments combining section 53 will bedescribed.

Although the segment combining section 53 is not essential, the additionof this section allows segments to be combined in a most appropriatemanner, thus enabling optimum approximation with a smaller number ofsegments. Referring to FIG. 11, the three segments A, B and C (see part[I] of FIG. 11) may be combined in three ways: (A+B, C), (A+C, B) and(B+C, A). Where control information 511 instructs the segments combiningsection 53 to combine the segments into two segments, the segmentscombining section 53 performs calculation to find one of the threecombinations which produces a smallest prediction error.

An example shown in part [II] of FIG. 11 is a case where a smallestprediction error is obtained with the combination (B+C, A). In thisexample, the segment B+C is newly represented by B. The segmentscombining section 53 outputs segment image information 503 representingthe segments shown in part [II] of FIG. 11. The frame memory 57 outputsprediction segment image information 506 to the segments combiningsection 53.

Receiving the combined segment image information 503, the segmentboundary line approximating section 54 performs boundary lineapproximation in a manner as shown in part [III] of FIG. 11. FIG. 12illustrates the operation of the segment boundary line approximatingsection 54.

In FIG. 12, parts (1) to (8) show eight patterns retained by the segmentboundary line approximating section 54 and their manner of variation,that is, manner of level determination. The segment boundary lineapproximating section 54 approximates the received image by applying oneof the eight patterns to it. In the example of FIG. 11, the segmentboundary line approximating section 54 selects pattern (3) shown in FIG.12. That is, the segment boundary line approximating section 54 outputspattern [IV] in response to pattern [II] (see FIG. 11). Approximatedsegment image information 504 is input to the motion vector re-searchsection 55.

The operation of the motion vector re-search section 55 will bedescribed below with reference to FIG. 11.

The motion vector re-search section 55 has the same configuration as themotion vector detecting section 52. However, the motion vector re-searchsection 55 does not detect all the motion vectors again, but determinesa boundary line 508 by searching for proper prediction segment imagesagain such that a level adjustment is performed on an approximatedpattern that is obtained by moving the boundary in the directionindicated by arrows in part [V] of FIG. 11 and a resulting pattern iscompared with the prediction segment images 507. Thus, the segmentboundary line approximating section 55 outputs motion vectors 510 ofrespective segments A′ and B′ having the boundary line 508, the boundaryline information 508, and prediction block image information 534.

The prediction block image 534 means a block including a plurality ofsegments being processed, that is, a square block shown in part [VI] ofFIG. 11. The segment boundary line coding section 56 encodes thereceived segment boundary line information 508, and outputs segmentboundary line coded information 509.

As described above, the segment boundary line approximating section 54of this embodiment retains the eight patterns shown in FIG. 12: ahorizontal variation of pattern (1), a vertical variation of pattern(2), a top-left/bottom-right variation of pattern (3), atop-right/bottom-left variation of pattern (4), a variation of pattern(5) in which an inside rectangular segment expands toward or contractsaway from the four corners, a variation of pattern (6) in which aninside circular segment expands toward or contracts away from the fourcorners, a variation of pattern (7) in which a square segment varies inthe top-left/bottom-right direction, and a variation of pattern (8) inwhich a square segment varies in the top-right/bottom-left direction.One of the above patterns is output as an approximation of the receivedsegment images 503.

Thus, the segment boundary line approximating section 503 need notperform fast calculation on the segment images 503 to determine theirshapes, and it can select a pattern and a level which provide a smallestprediction error simply by comparing the segment images 503 with theeight patterns.

FIGS. 13A and 13B, illustrate a method of coding a segment boundary linein this embodiment.

FIG. 13B shows a pattern that is finally selected by the motion vectorre-search section 55, which corresponds to pattern (3) of FIG. 12.

In FIG. 13A, “Pattern” indicates one of patterns (1) to (8) shown inFIG. 12 and “Level” indicates the position of a boundary line shown inpart [VI] of FIG. 11 or FIG. 13B. For example, a level value of 3 meansa boundary line close to the top-left corner, and the boundary linemoves toward the bottom-right corner as the level value increases (FIG.13B shows a case of a level value of 6. The segment boundary line codingsection 56 encodes the level value and outputs a resulting code as avalue of “Level” (see FIG. 13A). That is, “Pattern” and “Level”constitute the output signal 509.

On the other hand, “dmv1” and “dmv2” indicate the motion vectors 510.For example, “dmv1” and “dmv2” indicate motion vectors of the hatchedportion and the white portion in FIG. 13B.

Thus, where the number of segments is set at 2, “CODE(1)” indicatingthat segmented coding has been performed and “dmv1,” “dmv2,” “Pattern”and “Level” are transmitted. Where the conventional prediction coding isperformed on a block-by-block basis instead of the prediction codingwith segment-by-segment motion compensation, “CODE(0)” and the singlemotion vector of “dmv1” are transmitted.

According to the coding scheme described above, the number of bits ofthe segment boundary line coded information 509 is determined by thenumber of patters and the number of levels of the patterns (see FIG.12). In the example of FIG. 12 including eight patterns, “Pattern” isexpressed by 3 bits and “Level” is expressed by 5 bits if the number oflevels is 30.

The above embodiment is directed to the case where the segmentscombining section 53 is instructed by the control signal 511 to producetwo segments.

Alternatively, segment combining can be performed without input of thecontrol signal 511 by employing a scheme in which an evaluation valuethe segment combining is defined in advance, a combination allowableerror limit is determined, and the segment combining is effected if theevaluation value is smaller than the combination allowable error limit.That is, segments are combined if the evaluation value of theircombination is smaller than the sum of a prediction error and theallowable error limit.

As another alternative, there may be employed a scheme in which thecoding bit amount or the coding bit rate is input as the control signal511 and segment combining is effected if it exceeds a certain threshold.In this case, segments are combined when the bit rate has an enoughmargin, and are left separated when the bit rate does not have enoughmargin. Thus, the degree of prediction approximation can be improved.

Embodiment 5

This embodiment is directed to a case where a segmenting section iscomposed of a plurality of sections that operate according to differentschemes.

FIG. 14 shows the configuration of a segmenting section according tothis embodiment. In FIG. 14, a segmenting section 58 utilizes acharacteristic quantity 1, for instance, brightness. A segmentingsection 59 utilizes a characteristic quantity 2, for instance, a colordifference. A segmenting section 60 utilizes a characteristic quantity3, for instance, edge information. Reference numerals 520-522 denotesegment images produced by the respective segmenting sections 58-60.

The segment images 58-60 are subjected to evaluation of a predictionerror by using the segment boundary line information 508 and the motionvectors 510, which are outputs of the downstream motion vector re-searchsection 55. As a result, one of the segmenting sections 58-60 whichproduces a smallest prediction error is selected, that is, one of theiroutput signals 520-522 is selected as segment images 502, which aresupplied to the downstream sections. An evaluation section for selectinga segmenting section that provides a smallest prediction error may beprovided in the motion vector re-search section 55 or in a processorthat is separate from the configuration of FIG. 1.

Embodiment 6

This embodiment is directed to still another configuration of asegmenting section.

FIG. 15 shows the configuration of a segmenting section according tothis embodiment and related components. In FIG. 15, reference numeral 76denotes a frame buffer for storing a one-frame image; 61, a segmentingsection for dividing a one-frame image into a plurality of segmentimages; 62, a blocking section for dividing each of the segment imagesinto block images. On the other hand, reference numeral 62′ denotes ablocking section for dividing an input image into block images, andnumeral 63 denotes each of the block images into segment images.Reference numeral 544 denotes segment images of one frame, and numeral545 represents an original image of one frame.

The segmenting section 51 of the first embodiment corresponds to thecollection of the first set of components from the segmenting section 61to the blocking section 62 and the second set of components from theframe buffer 76 to the segmenting section 63.

The operation of the above configuration will be described below.

An important feature of the above configuration is the provision of thefirst set of components in which an input one-frame image is generallydivided into segment images 530 by the segmenting section 61 and theneach of the segment images 530 is further divided into block images 532by the blocking section 62. Segmented block images 532 (outputinformation) thus obtained are different from segmented block images 533that are obtained by dividing an one-frame image into block images 531and further segmenting each of the block images 531 in the same manneras in the fourth embodiment.

The former operation (first set of components) is effective inextracting a general structure or segment, while the latter operation(second set of components) is effective in extracting a finer segmentwithin a block. The output information is supplied to the segmentboundary line approximating section 54 and the motion vector re-searchsection 55, where segment boundary line information and motion vectorsare obtained. Then, as in the case of the fifth embodiment, one twokinds of output information which provides a smaller prediction error isselected in consideration of the coding bit amount or the coding bitrate. That is, one of the segmented block image information 532 and 533is selected to become segment image information 502.

FIG. 16 illustrates the operation of the components downstream of thesegmenting section. In FIG. 16, in the motion vector detecting section52 for determining a motion vector of a segment image A, a rectangulararea having apices P1-P4 is searched to detect a motion vector thatprovides a smallest prediction error. In this case, the movement area ofpoint P that is located in the segment image A is (X−DX, Y−DY) to (X+DX,Y+DY) where DX and DY are search widths. A motion vector 512 thusdetected is denoted by MV(MVx, MXy).

In the motion vector re-search section 55, a motion vector of anapproximated segment image A″ obtained by the segment boundary lineapproximating section 54 can be obtained by searching a smaller areahaving search widths (dx, dy) with the segment image A″ shifted by themotion vector 512 (dx<DX, dy<DY). The movement area of point Q that islocated in the segment image A″ is a rectangular are defined by Q1-Q4,whose coordinates are shown in FIG. 16.

Since the subsequent operation is the same as in the fourth embodiment,it is not described here.

Embodiment 7

This embodiment is directed to an image coding apparatus to which themotion prediction circuit of the fourth embodiment is applied.

FIG. 17 shows the configuration of a coding apparatus according to thisembodiment, in which a segment processing section 64 is thesegmentation-type motion prediction circuit of FIG. 10.

The following components shown in FIG. 17 are standard components usedin a common coding apparatus. Reference numeral 69 denotes a subtracterfor calculating a difference between an input signal 501 and an outputsignal 534 of the segment processing section 64. Numeral 65 denotes atransform coding section for converting prediction error block images535 (i.e., output of the subtracter 64) into transform coefficients 536.Numeral 66 denotes a quantizing section for quantizing the transformcoefficients 536 into quantized coefficients 537. Numeral 67 denotes adequantizing section for dequantizing the quantized coefficients 537.Numeral 68 denotes an inverse transform decoding section for returningan output 538 of the dequantizing section 67 to prediction error blockimages 539. Numeral 70 represents an adder for adding together aprediction image 541 (i.e., output of a frame memory 57) and theprediction error block images 539. For prediction of an image of thenext frame, an output 540 of the adder 70 is input to the frame memory57 as image information of the preceding frame. An output of the framememory (i.e., prediction image 541) is used in the segment processingsection 64.

On the other hand, reference numeral 71 denotes a variable-length codingsection for converting the quantized coefficients 537 (i.e., output ofthe quantizing section 66) into variable-length codewords 542. Numeral72 denotes a buffer for storing the variable-length codewords 542 andtransmitting output variable-length codewords 543 to a transmissionline.

The operation of the above image coding apparatus will be describedbelow.

The segment processing section 64 produces prediction block images 534(see FIG. 10) based on an input signal 501 and reference imageinformation 541 that is sent from the frame memory 57. The predictionsegment images 541 include prediction segment images 505-507 (see FIG.10). Then, a difference between the prediction block images 534 and theinput signal 501 is converted by the transform coding section 65 intotransform coefficients, which are converted into quantized coefficients537 by the quantizing section 66. On one hand, the quantizedcoefficients 537 are converted into prediction error block images 539through the dequantizing section 67 and the inverse transform decodingsection 68. The prediction error block images 539 are added to theprediction image 541 of the preceding frame, to become decoded blockimages 540, which is again stored into the frame memory 57.

On the other hand, the quantized coefficients 537 are subjected tovariable-length coding, to become variable-length codewords 542. Afterbeing stored in the buffer 72, the variable-length codewords 542 aretransmitted to the transmission line as variable-length codewords 543.As described in the fourth embodiment, the storage information amount ofthe buffer 72 is fed back, as the control signal 511, to the segmentprocessing section 64, more specifically, the segments combining section53, and used to control combining of segments within a block. Segmentboundary line coded information 509 and motion vectors 510 of therespective segments are transmitted from the segment processing section64 to the transmission line.

Embodiment 8

This embodiment is directed to an image decoding apparatus, whichcorresponds to the image coding apparatus of the seventh embodiment.

FIG. 18 shows the configuration of a decoding apparatus according tothis embodiment.

Important features of this embodiment reside in a segment boundary linedecoding section 74 and a prediction image generating section 75. Thefollowing components are equivalent to components of a common decodingapparatus. That is, reference numeral 73 denotes a variable-lengthdecoding section; 67, a dequantizing section; 68, an inverse transformdecoding section 68; 70, an adder; and 57, a frame memory.

The operation of the above decoding apparatus will be described below.

Variable-length codewords 543 received from a transmission line aresubjected to variable-length decoding in the variable-length decodingsection 73, to produce quantized coefficients 537. The quantizedcoefficients 537 are dequantized by the dequantizing section 67, toproduce transform coefficients 538, which are then converted intoprediction error block images 539 by the inverse transform decodingsection 68. The prediction error block images 539 are added toprediction block images 534 of the preceding frame by the adder 70, tobecome new decoded block images 540, which are stored into the framememory 57.

On the other hand, segment boundary line coded information 509 separatedfrom the reception signal is input to the segment boundary line decodingsection 74. The section 74 performs a converting operation the reverseto the operation of the segment boundary line coding section 56 of thefourth embodiment, to produce a segment boundary line 508 be decoding.Thus, the state of part [VI] of FIG. 11 is restored. The predictionimage generating section 75 generates the prediction block images basedon the segment boundary line 508 and motion vectors 510 that have beenseparated from the reception signal by referring to the predictionsegment images 541 of the preceding frame that are read from the framememory 57. Thus, fine segment images are obtained for each block.

Embodiment 9

This embodiment is directed to another segmentation-type motionprediction circuit.

Referring to FIG. 10 (fourth embodiment), this embodiment is intended tosimplify the operation of the motion vector re-search section 55 byutilizing motion vectors 512 detected by the motion vector detectingsection 52. That is, segment images newly approximated by the segmentboundary line approximating section 54 are similar in imagecharacteristics to segment images before the approximation, motionvectors are determined by referring to images in the vicinity of thesegment images before the approximation rather than referring to all theimages again. This operation shortens the search time.

According to the first aspect of the invention (first to thirdembodiments), an input image is divided into segments in accordance withits content and the segments are classified into classes in accordancewith the degree of importance in visual recognition. In coding eachsegment, the coding control parameter or coding scheme is changed inaccordance with its class so that the code amount of a segment that isless important in visual recognition is reduced. As a result, the amountof code data can be reduced while sufficient image quality is maintainedfor segments that are important in visual recognition.

In the configuration in which a prediction source image signal is readfrom a memory corresponding to the class indicated by a classidentification signal, the time required for the search of a predictionsource signal can be reduced.

If the coding time interval of each segment is changed in accordancewith its class and the memory updating interval is changed accordingly,the amount of code data as well as the load of memory rewriting can bereduced, contributing to increase of the coding efficiency.

According to the second aspect of the invention (fourth to ninthembodiments), since the motion prediction circuit has the segmentingmeans, the motion vector detecting means, the segment boundary lineapproximating means, and the motion vector re-search means, input blockimages can be transmitted or output as a smaller amount of imageinformation efficiently and with a small prediction error.

By virtue of the provision of the segments combining means, block imagescan be combined more efficiently into image information.

In combining segments, the coding bit amount or the coding bit rate isalso referred to. For example, the segment combining is effected whenthe coding bit rate is low, and is not effected when the coding bit rateis high. Therefore, high-quality image information can be obtainedwithin the limit of the transmission rate.

If a plurality of segmenting sections having methods of extractingdifferent features of an input image signal are provided and one ofthose segmenting sections is selected which provides the smallestprediction error, high-quality image information can be obtained.

The segmenting means is so constructed that selection is made betweenthe first segmenting section in which an input image signal is dividedinto blocks first and a feature of the input image signal is extractedon a block-by-block basis and the second segmenting section in which aninput image signal is divided into segments first by extracting afeature of the input image signal on a frame-by-frame basis and theneach segment is divided into blocks. Therefore, high-quality imagesignal can always be transmitted or output efficiently without beingaffected by a feature or a characteristic of the input image.

In the motion vector re-search operation, since final motion vectors aredetermined by referring to motion vectors detected by the motion vectordetecting means, the final motion vectors can be obtained in a shorttime.

According to the image coding apparatus, an image signal can be coded athigh prediction efficiency to produce a smaller amount of coded imagedata.

According to the image decoding apparatus, a smaller amount of codedimage data can be decoded at high prediction efficiency.

The invention being thus described it will be obvious that the same maybe varied in many ways. Such variations are not to be regarded as adeparture from the spirit and scope of the invention, and all suchmodifications as would be obvious to one skilled in the art are intendedto be included within the scope of the following claims.

1. A method for coding a prediction error signal, comprising: reading aprediction source image signal and an image signal to be coded, theimage signal having an arbitrary shape and size and varies in shape andsize from the prediction source image signal; interpolating orextrapolating the prediction source image signal within the image frameof the prediction source image signal to obtain a processed predictionsource image signal; generating a prediction signal based on theprocessed prediction source image signal; obtaining a prediction errorsignal based on the prediction signal and the image signal to be coded;and predictively coding the prediction error signal.
 2. An image codingapparatus, comprising: a memory storing a prediction source image signaland an image signal to be coded, the image signal having an arbitraryshape and size and varies in shape and size from the prediction sourceimage signal; a prediction section interpolating or extrapolating theprediction source image signal within the image frame of the predictionsource image signal to obtain a processed prediction source image signaland generating a prediction signal based on the processed predictionsource image signal; a prediction error extractor obtaining a predictionerror signal based on the prediction signal and the image signal to becoded; and a coder predictively coding the prediction error signal.